Skip to content

Measure every estimator, and fix what that found - #508

Merged
godofecht merged 6 commits into
mainfrom
estimator-coverage
Sep 27, 2026
Merged

godofecht merged 6 commits into
mainfrom
estimator-coverage

Conversation

@godofecht

Copy link
Copy Markdown
Owner

The canonical benchmark races 12 estimators. lib/scikit exports 203, so a claim about Flow against scikit-learn covered six percent of the library. This measures the rest, and fixes three real defects that only became visible once it did.

Coverage

bucket count
runnable, raced against a named scikit-learn class 172
different_shape: takes a pipeline, a vectorizer input or a list of fitted models first 25
flow_only: scikit-learn has no equivalent 6
total exported 203

Nothing is dropped silently. Every non-raced entry carries its reason in estimator_coverage.json.

Four pieces, all driven off one registry so the two sides cannot drift:

  • estimator_coverage.py parses every exported *_fit, resolves arguments from a table keyed by parameter name, maps to scikit-learn, and buckets each one.
  • generate_estimator_bench.py emits the Flow timing blocks, split across nine files because issue Config comparator flags identical settings expressed in different vocabularies #469 miscompiles large programs.
  • bench_estimators_sklearn.py times the other side. All 172 time cleanly.
  • compare_estimators.py joins them.

Defects it found

A heap-buffer-overflow in _solve_lstsq_qr. It assumed a design has at least as many rows as columns. RANSAC reaches it by fitting 5 sampled rows against the 10 features of diabetes. Three loops ran to n when only min(m, n) pivots exist, and both callers sized the solution buffer by rows while reading it back by columns. ASan named it; the process usually survived the bad read and died later with a varying exit code.

Eight models freed memory they did not own. local_outlier_factor, gaussian_process_classifier, nu_svr, sparse_coder, one_class_svm, radius_neighbors_classifier and both voting estimators stored a matrix or array straight from a fit parameter and released it in _free. Fitting and freeing the same data twice destroyed the caller's input. Found because the harness times over repeated fits; a single fit never touches the freed memory again.

svc was 4.8 seconds against scikit-learn's 0.6 ms, the worst row by three orders of magnitude. _smo_train recomputed every error term from scratch: an O(n) sum per candidate, inside a scan over all candidates to pick the partner, inside a pass over all n, for up to max_iter passes. It now uses the second-order working set selection that _kernel_svc_smo_precomputed already used for KernelSVC.

before after
svc fit 4833 ms 0.15 ms
svc row 0.0001x 2.6x

Accuracy on iris is unchanged at 0.9667 and test_opt_svc_predict still agrees with its independent per-pair reference to 1.8e-05.

Measurement

A single fit does not measure an estimator that finishes in microseconds. StandardScaler came back at 0.001 ms, which is the timer's resolution, and spectral_biclustering produced a 23600x ratio out of the same rounding; 23 rows were unrankable. Each block now probes once, picks a repeat count, and times over that many iterations. Two rows are still unranked and deserve it: kernel_density and nearest_neighbors do no work at fit time.

148 of the 170 ranked rows go to Flow. Apple M4 Max, Flow at -O3, fastest of three rounds, against the scikit-learn 1.9.0 wheel. The machine was not quiet, so these are indicative rather than publishable; CI stays the authority.

Read the result for what it is, and the artifact says so in its own contract field: no parity contract, no declared tolerances, each library on its own defaults over the same data. It answers whether an implementation is in the same performance league and says nothing about whether it computes the same thing. That stays the canonical benchmark's job.

The remaining losses cluster where scikit-learn has specialised solvers: the cross-validated linear paths (elastic_net_cv, multitask_lasso_cv), the manifold embeddings (mds, lle, spectral_embedding), and the SVM variants still on the old dense solver (nu_svr, svr, one_class_svm).

Verification

Full suite 114/114, run twice: once after the SVC change and again after the ownership fixes. ASan clean on the case that found the overflow. All nine generated files exit 0 under repeated fits, where five crashed before the ownership fixes. Evidence validators green.

RANSAC samples min_samples rows and fits a linear regression on them. With
diabetes that is 5 rows against 10 features, and _solve_lstsq_qr assumed the
design has at least as many rows as columns. Three loops ran to n when only
min(m, n) pivots exist, and both callers sized the solution buffer by rows
while reading it back by columns.

AddressSanitizer on the generated estimator benchmark:

  ERROR: AddressSanitizer: heap-buffer-overflow
  READ of size 8 in _solve_lstsq_qr
    #1 linear_regression_fit
    #2 ransac_regressor_fit

The process survived the read often enough to print its result and then died
during cleanup, with the exit code varying between runs, which is the heap
corruption signature AGENTS.md describes.

The solver now runs its sweeps to the number of pivots that exist and zeroes
the coordinates the data does not determine. The two callers allocate the
solution buffer as max(rows, features). ASan is clean on the case that found
it, and the RANSAC row now runs to completion.
The canonical benchmark races 12 estimators. lib/scikit exports 203, so a
statement about Flow against scikit-learn covered six percent of the library.

estimator_coverage.py parses every exported *_fit, resolves its arguments from
a table keyed by parameter name, finds the scikit-learn class, and sorts each
one into a bucket with a recorded reason:

  runnable         172   raced against a named scikit-learn class
  different_shape   25   takes a pipeline, a vectorizer input or a list of
                         fitted models first, so it needs its own harness
  flow_only          6   scikit-learn has no equivalent

generate_estimator_bench.py emits the Flow timing blocks from that registry,
split across nine files because issue #469 miscompiles large programs, and
flushes after each line. Without the flush one estimator trapping took the
whole file's buffered output with it, which is how a crash first presented as
twenty silently missing rows. flow run reports exit 0 for a process killed by
a signal, so the harness checks the binary's own exit code.

bench_estimators_sklearn.py times the other side of the same registry and
compare_estimators.py joins them. All 172 time cleanly on both sides.

Read the result for what it is, and the artifact says so in its contract
field: no parity contract, no declared tolerances, each library on its own
defaults over the same data. It answers whether an implementation is in the
same performance league and says nothing about whether it computes the same
thing. That stays the canonical benchmark's job.

On an Apple M4 Max with Flow at -O3, fastest of five rounds against the
scikit-learn 1.9.0 wheel, Flow wins 137 of the 170 rows that produce a ratio.
The machine was not quiet, so these are indicative rather than publishable;
CI remains the authority. kernel_density and nearest_neighbors do no work at
fit time and record no ratio rather than a fabricated one.

The widest loss is svc at 0.0001x, 4.8 seconds against 0.6 ms on iris. It
calls _smo_train, which recomputes every error term with an O(n) inner loop on
every pass of every one of up to 1000 iterations. The library already contains
the solver that fixed this for KernelSVC, which maintains the gradient
incrementally and converges in about sixty steps.
The wide estimator benchmark put svc at 0.0001x: 4.8 seconds against
scikit-learn's 0.6 ms on iris, the worst row by three orders of magnitude.

_smo_train recomputed every error term from scratch. An O(n) sum per
candidate, inside a scan over all candidates to pick the partner, inside a
pass over all n, for up to max_iter passes. On 150 samples that is billions of
operations for a problem libsvm settles in tens of steps.

It now maintains the gradient incrementally and selects the working set by
second order information, which is WSS3 of Fan, Chen and Lin 2005 and what
_kernel_svc_smo_precomputed has been doing for KernelSVC. The kernel is
already materialized here, so rows come straight off K.data rather than
through matrix_at, which copies the Matrix struct per element. Since y squared
is one, the curvature terms reduce to plain kernel entries.

svc fit goes from 4833 ms to 0.15 ms, and the row from 0.0001x to about 2.5x.
Accuracy on iris is unchanged at 0.9667 and test_opt_svc_predict still agrees
with its independent per-pair reference to 1.8e-05.

The bias comes from the free support vectors. Where a fit leaves none of them,
every support vector sits at a bound and their average is the best estimate
available, which is the case the previous code could not reach because it
tracked b as it went.
Eight fitted models stored a matrix or an array straight from a fit parameter
and then released it in their free function. Fitting and freeing the same data
twice therefore destroyed the caller's input and read it back:

  ERROR: AddressSanitizer: heap-use-after-free
  READ in matrix_at
    #1 local_outlier_factor_fit
  freed by matrix_free
    #1 local_outlier_factor_free

The wide estimator benchmark found this by timing each estimator over repeated
fits. A single fit followed by a single free never touches the freed memory
again, so the defect was invisible to every caller that fits once.

local_outlier_factor, gaussian_process_classifier, nu_svr, sparse_coder,
one_class_svm and radius_neighbors_classifier now copy what they keep, which
is what KernelSVC and SVC already did. voting_classifier and voting_regressor
copy their classes and weights arrays.

Both voting estimators still take over the fitted sub-estimators handed to
them, because their free functions release those too and changing that would
move ownership rather than fix a bug. That contract is now written down where
the field is assigned.
A single fit does not measure an estimator that finishes in microseconds.
StandardScaler on 150 rows by 4 came back as 0.001 ms, which is the timer's
resolution rather than its cost, and spectral_biclustering produced a 23600x
ratio out of the same rounding. Twenty three rows were unrankable.

Each block now probes once, picks a repeat count from what it sees, and times
fit and predict over that many iterations. Two rows are still unranked, and
both deserve it: kernel_density and nearest_neighbors do no work at fit time.
The comparison refuses to rank anything left at the floor rather than print a
ratio it cannot support.

The repeats paid for themselves immediately by crashing five of the nine
files, which is how the aliased model ownership fixed in the previous commit
came to light. Fitting once and freeing once never touches the freed memory
again.

148 of the 170 ranked rows go to Flow, with svc now at 2.6x rather than
0.0001x. The losses cluster where scikit-learn has specialised solvers: the
cross-validated linear paths, the manifold embeddings, and the SVM variants
that still use the old dense solver.
The benchmarks page gains the wide matrix: 166 estimators ranked against their
scikit-learn counterpart, sorted narrowest first, with the counts for the ones
that are not raced and the reason each is excluded. index.html names coverage
as its own link in the evidence chain.

The page says plainly what these rows leave out. No parity contract, no
declared tolerances, no disparity report, each library on its own defaults,
timed on a developer machine that was not idle. The canonical rows above
remain where numerical equivalence is established.

Four estimators are excluded from the count because their own comments call
them simplified, and the registry detects that from the source rather than
from a list. spectral_biclustering thresholds row and column means where
scikit-learn does an SVD and k-means; at 25106x it was the widest number on
the page and it was measuring two different algorithms. spectral_coclustering
and elliptic_envelope were second and third. Excluding them also removes nca
from the loss column, so the filter cuts both ways.

The page also warns about the ratios it does publish. The two sides run their
own defaults, so a row can differ by three orders of magnitude because of how
much work each one does. Iris carries no missing values, which leaves the
imputers nothing to impute.
@godofecht
godofecht merged commit 1bb7f71 into main Sep 27, 2026
10 checks passed
@godofecht
godofecht deleted the estimator-coverage branch September 27, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant